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Abstract 

Substantial progress has been made recently towards design- 
ing, building and test-flying remotely piloted Micro Air \ chicles 
(MA Vs). We seek to complement this progress in overcoming the 
aerodynamic obstacles to flight at very small scales with a vision- 
guided flight stability and autonomy system. The developed sys- 
tem is based on a robust horizon detection algorithm which we 
discuss in greater detail in a companion paper [5]. In this paper, 
we first motivate the use of computer vision for MA V autonomy, 
arguing that given current sensor technology \ vision may be the 
only practical approach to the problem. We then briefly review 
our statistical vision-based horizon detection algorithm, which 
has been demonstrated at 30Hz with over 99.9% correct horizon 
identification. Next, we develop robust schemes for the detection 
of extreme MA V attitudes, where no horizon is visible, and for the 
detection of horizon estimation errors, due to external factors 
such as video transmission noise. Finally, we discuss our feed- 
back controller for self-stabilized flight, and report results on vi- 
sion-based autonomous flights of duration exceeding ten minutes. 

1. Introduction 

Ever since humankind’s first powered flight, research efforts 
have continually pushed the envelope to create flying machines 
that are faster and/or larger than ever before. Now, however, there 
is an effort to design aircraft at the other, largely unexplored end 
of the spectrum, where the desire for portable, low-altitude aerial 
surveillance has driven the development and testing of aircraft 
that are as small and slow as the laws of aerodynamics will permit 
— in other words, on the scale and in the operational range of 
small birds. Vehicles in this class of small-scale aircraft are 
known as Micro Air Vehicles or MA Vs. 

Equipped with small video cameras and transmitters, MAVs 
have great potential for surveillance and monitoring tasks in areas 
either too remote or too dangerous to send human scouts. Opera- 
tional MAVs will enable a number of important missions, includ- 
ing chemical/radiation spill monitoring, forest-fire 
reconnaissance, visual monitoring of volcanic activity, surveys of 
natural disaster areas, and even inexpensive traffic and accident 
monitoring. Additional on-board sensors can further augment 
MAV mission profiles to include for example airborne chemical 
analysis. 


In the military, one of the primary roles for MAVs will be as 
small-unit battlefield surveillance agents, where MAVs can act as 
an extended set of eyes in the sky for military units in the field. 
This use of MAV technology is intended to reduce the nsk to mil- 
itary personnel and has, perhaps, taken on increased importance 
in light of the U.S.’s new war on terrorism, where special opera- 
tions forces will play a crucial role. Virtually undetectable from 
the ground, MAVs could penetrate potential terrorist camps and 
other targets prior to any action against those targets, significantly 
raising the chance for overall mission success. 

Researchers in the Aerospace Engineering Department at the 
University of Florida have established a long track record in de- 
signing, building and test-flying (remotely human-piloted) practi- 
cal MAVs [7-9,14,15]. While much progress has been made in the 
design of ever smaller MAVs by researchers at UF and others in 
the past five years, significantly less progress has been made to- 
wards equipping these MAVs with autonomous capabilities that 
could significantly enhance the utility of MAVs for a wide array 
of missions. 

The first step in achieving such MAV autonomy is basic sta- 
bility and control. Here, we present such a flight stability and con- 
trol system , based on vision processing of video from a camera 
on-board our MAVs. In this paper, we first motivate the use of 
computer vision for such a control system, and briefly review our 
related work in vision-based horizon detection which is discussed 
more fully in a companion paper [5], and forms the basis of the 
flight stability system presented here. Next, we address real-time 
control issues in the flight stability system, including extreme at- 
titude detection (i.e. no horizon in the image), confidence mea- 
sures for the detected horizon estimates, and filtering of horizon 
estimates over time. Finally we report some results of self-stabi- 
lized MAV flights over the campus of the University of Florida 
and over Fort Campbell, Kentucky. 

2. Horizon detection 

Below, we motivate vision-based approaches to the flight sta- 
bility and control problem for MAVs and review related w'ork in 
in vision-based horizon detection. 

2.1 Motivation 

MAV flight stability and control presents some difficult chal- 
lenges. The low moments of inertia of MAVs make them vulner- 
able to rapid angular accelerations, a problem further complicated 
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by the fact that aerodynamic damping of angular rates decreases 
with a reduction in wingspan. Another potential source of insta- 
bility for MAVs is the relative magnitudes of w ind gusts, which 
are much higher at the MAV scale than for larger aircraft. Ln fact, 
wind gusts can typically be equal to or greater than the forward 
airspeed of the MAV itself. Thus, an average wind gust can im- 
mediately affect a dramatic change in the flight path of these ve- 
hicles. 

Birds, the biological counterpart of mechanical MAVs, can of- 
fer some important insights into how one may best be able to 
overcome these problems. For aerospace and robotic researchers, 
the extraordinary capabilities of birds are a source of wonderment 
and frustration at the same time — wonderment, because birds ex- 
hibit a remarkably complex and rich set of behaviors, frustration, 
because the duplication of those behaviors in man-made systems 
has thus far been elusive. Given this sad state of affairs, it is nat- 
ural for engineers and researchers to want to learn from and emu- 
late these biological systems. Here, we do not intend to seek 
structural equivalence between artificial MAV systems and bio- 
logical neural systems; rather, we seek to learn important func- 
tional lessons from biology. In studying the nervous system of 
birds, one basic observation holds true for virtually all of the thou- 
sands of different bird species: Birds rely heavily on sharp eyes 
and vision to guide almost every aspect of their behavior [5,1 6]. 

Biological systems, while forceful evidence of the importance 
of vision in flight, do not, however, in and of themselves warrant 
a computer- vision based approach to MAV autonomy. Other 
equally important factors guide this decision as well. Perhaps 
most critical, the technologies used in rate and acceleration sen- 
sors on larger aircraft are not currently available at the MAV 
scale. It has proven very difficult, if not impossible, to scale these 
technologies down to meet the very low r payload requirements of 
MAVs. While a number of sensor technologies do currently exist 
in small enough packages to be used in MAV systems, these small 
sensors have sacrificed accuracy for reduced size and weight. 
Take, for example, MEMs (Micro Electro-Mechanical Systems) 
rate gyros and accelerometers. MEMs piezoelectric gyros, while 
only weighing approximately one gram, have drift rates on the or- 
der of 100° per minute and are highly sensitive to changes in 
temperature. While elaborate temperature calibration procedures 
can improve their accuracy somewhat, their use in inertial naviga- 
tion is problematic at best. 

Even if sufficient rate and acceleration sensors did exist, how- 
ever, their use on MAVs may still not be the best allocation of 
payload capacity. For many potential MAV missions, vision may 
be the only practical sensor than can achieve required and/or de- 
sirable autonomous behaviors. Furthermore, given that surveil- 
lance has been identified as one their primary missions, MAVs 
must necessarily be equipped with on-board imaging sensors, 
such as cameras or infrared arrays. Thus, computer- vis ion tech- 
niques exploit already present sensors, rich in information con- 
tent, to significantly extend the capabilities of MAVs, without 
increasing the MA V’s required payload. 

2.2 Vision-based horizon detection 

Fundamentally, flight stability and control requires measure- 
ment of the MAV’s angular orientation. While for larger aircraft 
this is typically estimated through the integration of the aircraft’s 


angular rates or accelerations, a vision-based system can directly 
measure the aircraft’s orientation with respect to the ground. The 
two degrees of freedom critical for stability — the bank angle $ 
and the pitch angle 0 1 — can be derived from a line correspond- 
ing to the horizon as seen from a forward facing camera on the air- 
craft. In [5], we develop and report results for a vision-based 
horizon-detection algorithm, which attempts to minimize the in- 
tra-class variance of the ground and sky pixel distributions. Figure 
1 illustrates several examples of our current algorithm at work. In 
each image, the yellow line indicates the algorithm’s estimated lo- 
cation of the horizon. The algorithm has been demonstrated to run 
at 30 Hz on a 900 MHz x86 processor with image resolution of 
320 x 240 , although acceptable performance has been shown for 
image resolutions as low 160 x 120 . 

At different times of the day, and under both fair and cloudy 
conditions, we have gathered hours of video on-board our MAV, 
flying under manual control over terrain that includes roads, 
buildings large and small, meadows, w'ooded areas, and a lake. 
For these data, our horizon-detection algorithm correctly identi- 
fies the horizon in over 99.9% of cases. 

3. Flight stability and control 

In this section, we extend the basic horizon-detection algo- 
rithm to real-time horizon tracking. Below, w'e consider the fol- 
lowing important issues: (1) extreme attitude detection, (2) error 
detection in horizon estimation, (3) filtering of the horizon esti- 
mate overtime, and (4) basic feedback control and stabilization of 
the MAV. 

3.1 Extreme attitude detection 

One of the implicit assumptions of the horizon detection algo- 
rithm is that there will always be a horizon in the images from the 
forward looking camera on board the MAV. In real-time control 
of the MAV, the MAV may, however, encounter times when no 
visible horizon appears in the image, if, for example, a gust of 
wind forces the nose of the aircraft too far up or dow n. Such cases 
cannot simply be ignored; if the aircraft is heading straight to- 
wards the ground, no horizon will be visible in the camera image, 
yet the control system will certainly be required to take action to 
save the MAV from certain and possibly catastrophic crashing. 

It is desired then, to be able to detect instances when the hori- 
zon is not in view of the camera, and if so to determine what ac- 
tion to take in order to bring the horizon back into view. There are 
two valuable sources of information which we can draw' on to de- 
tect these types of extreme attitudes: (l) recent appearance of the 
sky and ground from previous time steps, and (2) recent location 
of the horizon line from previous time steps. For example, if the 
horizon line was recently estimated to lie near the top of the im- 
age, it is logical that a subsequent image without a horizon line is 
most likely a view of the ground. We can use these two pieces of 
information to quantitatively determine if the horizon line exists 
in the image and if not, to determine w’hether we are looking at the 
sky or the ground. 

1. Instead of the pitch angle 0 , we actually recover the closely 
related pitch percentage o in our horizon-detection algo- 
rithm. 
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Fig. 1: (a) Horizon detection example; (b) horizon detection 
with uneven horizon; (c) horizon detection with severe video 
interference. 


As part of our horizon detection algorithm, we already com- 
pute the aggregate statistical properties of the sky and ground in 
RGB color space. Let us denote all the sky pixels in an image as. 



where r- denotes the red channel value, g- denotes the green 
channel value and b* denotes the blue channel value of the i th 
sky pixel, and let us denote all the ground pixels in an image as. 



where rf denotes the red channel value, gf denotes the green 
channel value and bf denotes the blue channel value of the i th 
ground pixel. Then, as part of the horizon detection algorithm, we 
compute the following statistical descriptions of the sky and 
ground appearance: 
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where, \i s and |i denote the means of the sky and ground pixels, 
respectively, and Z 5 and denote the covariances of the sky 
and ground pixels, respectively. 

We now use the above statistics to model the appearance of the 
sky and ground over a recent time history of the MAV’s flight. 
Our general approach for detection of extreme attitudes keeps 
running statistical models for both the sky and ground from pre- 
vious frames, where horizon lines were detected with a high de- 
gree of confidence. With each new frame, the result of the horizon 
detection algorithm can be checked by comparing the sky and 
ground models for the current frame with the computed, time-de- 
pendent statistical models for sky and ground. If the distributions 
on either side of the line in the current frame both appear to be 
more similar to the known ground distribution, then it would ap- 
pear that the aircraft is pointing towards the ground. Conversely, 
if they both match the sky better, then it is advisable to nose 
downward. Interestingly, if the sky in the current frame matches 
the ground model while the ground in the current frame matches 
better with the sky model, we can detect situations where the 
plane is flying upside down. 

One additional piece of information is required to implement 
the extreme attitude detection scheme, namely, a time history of 
the horizon line estimate. For the purposes of detecting extreme 
attitudes, we are most concerned with a recent history of the pitch 
percentage cj , the percentage of the image below the horizon line. 
One measure of that history is a running average O avg of the pitch 
percentage over the previous ten frames. 

Upon startup of the system, the camera is assumed to be ori- 
ented such that the horizon is in its view. When the first frame of 
video is processed by the system, the means and covariance ma- 
trices of the ground and sky models are set equal to those found 
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by the horizon detection algorithm. The system then begins to up- 
date the models using the results of the horizon detection algo- 
rithm for a set number of initialization frames. Our current 
implementation uses 100 initialization frames (3.3 seconds). 
Once boot-strapped, it is necessary to continually update the sky 
and ground models as the aircraft flies to account for changes in 
lighting associated with changes in orientation and changes in 
landscape, etc. The running statistical models are updated as fol- 
lows: 


z s(l) = oS, (f) + (1-0)2, (6) 

E g(r) = al g(0 + ~ a ^g (T> 

M, (() = “M, ( ,) + (l-a)M, (8) 

= a %f) + (1 - a) ^ (9) 


where , Z ^ and are the time-dependent mod- 

el covariances and means, respectively, while Z g , Z g , \i s and 
\X g are the covariances and means for the current frame. Note that 
the constant a controls how rapidly the models change overtime. 

For a new image, we first compute th estimated horizon for 
that image. We then compare the resultant current statistics with 
the running statistical models from previous frames, using the fol- 
lowing four distance measures: 


d 3 = Vswfcslnteg- ^(o> + 
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The value of D ] measures the similarity between the region se- 
lected as the sky by the horizon detection algorithm in the current 
frame and the sky model from recent frames. D 2 represents the 
similarity between the currently computed sky region and the 
ground model from recent frames. Likewise, the values of D 3 
and Z ) 4 are the similarity measures between the current ground 
region and the sky and ground models from recent frames, respec- 
tively. Table 1 now summarizes four possible cases and the con- 
clusions we can draw for each case. 

The determinations in the above table can now be combined 
with the past history of the horizon line to decide what action to 
take. If the current frame is determined to be normal by the valid- 
ity test (case 1 ), then the horizon estimate is assumed to be accu- 
rate, and commands sent to the MAV are determined by the 
normal control system loop described in Section 3.4. Also, the 
statistics of the validated frame are used to update the sky and 
ground models per equations ( 6 ) through (9). If the validity test 
returns a higher likelihood of all ground (case 2 ), we verify that 
result with the recent history of the horizon line o avg to deter- 


Table 1 : Extreme attitude detection 


case 

condition 

conclusion 

1 

£>, < D 2 and D 3 > D 4 

valid horizon present 

2 

Dj >D 2 and D 3 > Z ) 4 

all ground 

3 

£)j < D 2 and Z ) 3 < Z ) 4 

all sky 

4 

Dj > D 2 and D 3 < D 4 

upside down 


mine what action to take. When the value of G avg is above a set 
threshold, then the system goes into a “pull-up” mode that sends 
commands to the aircraft to rapidly increase its pitch angle. A val- 
ue of 0.8 was used for this threshold. While the system is in pull- 
up mode, the time-dependent statistical models are not updated 
since the horizon estimate during this time will most likely be in- 
correct. Also during pull-up mode, a avjf is only updated with the 
estimated value of o if the validity test indicates the current 
frame has returned to a visible horizon line; otherwise, o avg is 
updated using a value of l .01. The system will stay in pull up 
mode until a valid horizon is detected. Similarly, if the validity 
test returns a higher likelihood of all sky (case 3) and the value of 
cy is below a given threshold (set at 0 . 2 ), the system goes into 
a “nose-down” mode. Updating of the time- dependent statistical 
models and a in nose-down mode is the same as in pull-up 
mode, except that the default update value for G avg is 0.01 in- 
stead of 1 . 0 ! . 

3.2 Error detection in horizon estimates 

Extreme attitude detection can also help us to detect possible 
errors in the horizon estimation algorithm; such errors can occur 
when transient noise causes video degradation. Consider, for ex- 
ample, the following possibility: the validity test returns case 2 
{all ground), but G aviJ < 0.8 . In this situation, we must assume an 
error occurred in horizon detection, because the aerodynamic 
characteristics of the plane do not permit such sharp changes in 
pitch over l/30th of a second. More generally, if the validity test 
returns any of the non-normal cases (2, 3 or 4) and the value of 
a does not conform to the appropriate threshold values, we 
consider the horizon detection for that frame to be in error. In this 
case, the horizon estimate from the previous frame is used to esti- 
mate the horizon parameters for the current frame. 

From extensive flight testing, we observer qualitatively that 
this extreme attitude and error detection system performs well. It 
is difficult to quantitatively assess the performance of the system 
on real-time data since there is no “correct” answ er with which to 
compare it. Both the qualitative viewing of the output, however, 
along with successful flight tests indicate that the system per- 
forms adequately. 

Initial tests of the system were performed by playing previous- 
ly recorded video of human-piloted MAV flights into the system. 
Figure 2 shows a sample of recorded output data from the system 
in which the video signal is corrupted by noise. The plot of pitch 
percentage estimates are overlaid onto the plot of the error detec- 
tion trigger. When an error is detected, the error detection trigger 
has a value of 1 . As can be seen in Figure 2, the algorithm uses the 
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Fig. 2: Pitch percentage estimates with video noise. 


last known good estimate during the time of the video transmis- 
sion error, instead of the estimates from the bad video frames. 

Figure 3 illustrates an example of extreme attitude detection. 
As soon as the horizon disappears off the top of the image, the er- 
ror detection trigger value jumps to 1. The controller then goes 
into pull-up mode until the horizon reappears near the top of the 
image. The pitch estimate remains at 1.01 until the controller exits 
the pull-up mode. 

3.3 Kalman filtering 

In order to make the horizon estimates usable for self-stabili- 
zation and control, the horizon estimates, after being processed by 
the extreme attitude and error detector, are passed through a Kal- 
man filter [1], The Kalman filter provides an optimal estimate of 
a system’s current state, given a dynamic system model, a noise 
model, and a time series of measurements. While a dynamic mod- 
el of the system is desirable, the formalism of the Kalman filter 
can be employed even without an accurate dynamic model. Since 
no dynamic model is readily available for our flexible-wing 
MAVs 1 , we model the system state (the two parameters of the ho- 
rizon estimate) as two simple first-order, constant-velocity sys- 
tems. As such, the Kalman filter has the effect of removing high 
frequency noise from the system measurements and eliminating 
any radical single frame errors not first caught by the error detec- 
tion system. The principal benefit of the Kalman filter for our ap- 
plication is that it effectively eliminates unnecessary small control 
surface deflections due to noise. 

3.4 Feedback control 

To date, we have employed a very simple controller to validate 
vision-based flight stability and control for MAVs. For simplicity, 
the bank angle <J> and pitch percentage o are treated as indepen- 
dent from one another, and for both parameters, we implement a 
simple PD (proportional/derivative) feedback control loop, with 
gains determined experimentally from flight tests; each control 
loop is updated at full frame rate (i.e. 30 Hz). In initial flight tests, 
the derivative gains were set to zero. 

/. A dynamic model for our MA V airframes is currently being 
developed at NASA Langley Research Center. With the code 
infrastructure in place , the dynamic model can be added when 
i t beco m es a vai la hie. 



Fig. 3: Extreme attitude detection. 


4. Self-stabilized flight 
4.1 Experimental setup 

Figure 4 illustrates our current experimental setup. The video 
signal from the MAV is transmitted from the plane through an an- 
tenna to a ground-based computer, where all vision processing is 
performed. In manual mode, the plane is controlled during flight 
by a remote human pilot through a standard radio link. In autono- 
mous mode, the plane is controlled through the feedback control- 
ler which sends control surface commands to the MAV through a 
custom designed interface over the same radio link. Our interface 
allows the PC to control a standard Futaba radio transmitter 
through an RS-232 serial port. 

The MAV used for test flights had a wingspan of 18 inches. 
While we have designed and flown MAVs with wing spans as 
small as seven inches, we selected the somewhat larger platform 
both for its increased dynamic time constants and its ability to car- 
ry a high-powered video transmitter (i.e. increased payload). Fig- 
ure 5 shows a picture of this MAV. The on-board camera is a 
monolithic CMOS type camera with a 1/3 inch sensor area, and is 



Fig. 4: Experimental setup. 




Fig. 5: MAV used for test flights. 


connected to an 80 mW video transmitter. The MAV is powered 
by electric propulsion and has differential elevons for control, al- 
though the software is written to support both elevon and rudder- 
elevator control designs. 

The PC interface uses a PIC microcontroller to translate serial 
commands from the PC into the pulse width modulated signals re- 
quired for input to the transmitter. A carbon fiber housing was 
constructed to hold the circuit board and port connectors for the 
interface. 

4.2 Flight testing 

Flight testing proceeds as follows. Prior to launch, the aircraft 
is oriented such that the horizon is in the field-of-view of the cam- 
era. This allows the algorithm to build initial models of the sky 
and the ground; while these models are not used in the horizon- 
detection algorithm itself [5], they are used for extreme attitude 
and error detection. 

Upon launch, flights are controlled by a human pilot until the 
MAV reaches sufficient altitude. At that point, control is trans- 


ferred to the automated flight control and stability system; in case 
of catastrophic failure (loss of video signal, etc.), the radio trans- 
mitter is equipped with an override button to allow the human pi- 
lot to regain control at any time if necessary. 

A joystick connected to the PC can be used to adjust the de- 
sired heading for the controller. The joystick input effectively 
commands a bank and pitch angle for the aircraft to follow. Later 
flights used a pre-programmed set of maneuvers for truly autono- 
mous flight. To date, we have flown uninterrupted autonomous 
flights of over 10 minutes, flights that ended only due to video 
transmission interference, or low on-board battery power. 

Figure 6 below illustrates a short, 90-second run of actual 
flight data, where the flight vehicle was under vision-guided con- 
trol above the University of Florida campus. 1 More recently, the 
same vision-based control system successfully flew over substan- 
tially different terrain at Fort Campbell, Kentucky. Videos of the 
current algorithm performance under manual and automated 
flight stabilization can be viewed at http://mil. ufj.edu/~nechvba/'' 
mav . 

5. Conclusion 

Qualitatively, even our simple PD control system provides 
much more stable control than that of our best human pilots, both 
in terms of steady, level flight, and in coordinated turns. Human 
pilots can typically not hold the plane on a steady, level heading 
for more than a few fractions of a second; under vision-guided 
control, however, we were able to fly long straight segments that 
were limited only by the range of the video transmitter (see Figure 
7 for a 7.5 second stretch of self-stabilized straight and level 
flight). Prior to the development of the horizon-tracking control 
system, only pilots with extensive training could learn to fly our 
micro air vehicles; with the automated control system, however, 
people who had never piloted any aircraft before were able to eas- 
ily guide the MAV above the flying arena. It is this fact alone that 
speaks the most to the potential value of this work. Ideally, one 

1. The data in Figure 6(h) appears more oscillatory than the 
actual flight, because of light non-linear coupling between 
pitch percentage and roll angle . 



Fig. 6: 90 seconds of sample flight data: (a) commanded roll angle (red) vs. vision-based measured roll angle (blue); (b) 
commanded pitch percentage (red), vs. vision-based measured pitch percentage. 
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Fig. 7: Image sequence of self-stabilized straight and level flight (7.5 seconds, images are 20 frames apart). Note that the black 
lines at the top of the image are the propeller. 


wants MAVs to be deployable by a wide range of people, not only 
expert RC pilots; while much remains to be done, including auto- 
mating landings and take-offs, the work in this paper is a big step 
towards the development and deployment of usable and practical 
MAVs. 
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